Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ssh folder pull #707

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

douglas-raillard-arm
Copy link
Collaborator

Fix SSH pull for folders over SFTP and asyncify the generic part of the push/pull implementation so it can run concurrently for multiple files when globbing is used.

Paramiko seems to have had a slight change in behavior that broke
devlib: to save a remote command execution, we attempt to pull any path
as file first, and then as a folder if the former failed.

This is now broken as paramiko will create an empty destination file
when trying to pull as a file. When we attempt again to pull as folder,
the destination exists already (empty file) and we raise an exception.

To fix that, make sure we cleanup any attempt after pulling as a file
before trying again.
When pulling a file from the target, copy all paths as files and follow
symlinks if necessary. That fixes issues related to chmod not working on
symlinks and generally allows getting any path.

If we want to one day preserve symlinks in some capacities, we can
always add an option to Target.pull() to do so.
Ensure we use the busybox command in operations involved in file
transfers.
* Combine cp and chmod for pull
* Make both push and pull use concurrent async code
_prepare_xfer() deals with all the paths resulting from glob expansion,
so it can benefit from async capabilities in order to process multiple
files concurrently.

Convert the internals to async/await to enable useful map_concurrently()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant